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Executive Summary 


The National Assessment of Educational Progress (NAEP) 2017 pilot study of the Middle 
School Transcript Study (MSTS) is the first NAEP transcript study to focus on the middle school 
level, collect student coursetaking data at the Trial Urban District Assessment (TUDA) level, and 
rely solely on the electronic submission of course catalogs and student transcripts. The purposes 
of this pilot study were to evaluate the feasibility of providing middle school student 
coursetaking data that link to the NAEP assessment data and contextual questionnaire data, to 
provide coursetaking measures and analyses that focus on issues relevant at the middle school 
level, and to explore the use of electronic school record collections as the data delivery system 
for future transcript studies. 


This pilot study, conducted in three major stages, included recruitment, data collection and 
course coding, and analysis and reporting. It was limited to large urban districts, referred to as 
the TUDA jurisdictions, to help evaluate and streamline operations for potential future NAEP 
TUDA-based middle or high school transcript studies, as well as other electronic data-based 
transcript studies. The participation in this study was strictly voluntary for TUDA jurisdictions. 
TUDA jurisdictions who volunteered to participate agreed to submit information for all eighth- 
grade students enrolled in the schools sampled for the NAEP 2017 TUDA-based mathematics 
and reading assessments. 


General Operational Findings of the Pilot Study 


Overall, the NAEP 2017 MSTS pilot study was successful. All participating TUDA jurisdictions 
could submit required data files electronically and were able to provide course IDs on both the 
course catalogs and student course records, which made coding course records more efficient. 


Below are specific operational findings: 


e Approximately 123,800 student course records were collected, and 107,900 records were 
analyzed for the NAEP 2017 MSTS pilot study. 


e Analyses on coursetaking measures (i.e., credits earned and grade point averages) showed 
similar general patterns in relationships that have been illustrated in the previous 
iterations of the NAEP High School Transcript Study (HSTS). 


e Analyses on coursetaking measures and NAEP assessment scores demonstrated a 
successful linkage of student coursetaking information with the NAEP assessment and 
contextual questionnaire data, and showed similar general patterns in relationships among 
coursetaking, assessment and contextual data that had been illustrated in the prior NAEP 
HSTS. 
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Summary of Analytic Results 


In examining the feasibility of providing student coursetaking data to supplement the NAEP 
assessment data, several analytic inquiries were conducted to investigate student coursetaking 
measures. These inquiries included how coursetaking measures relate to student performance on 
NAEP assessments, how state assessments of student achievement relate to NAEP assessment 
scores, as well as how coursetaking measures relate to NAEP student contextual indices. 
Additional analyses of various academic programs (e.g., IB Middle Years Programme, and 
Project Lead the Way: Gateway) were conducted to further examine the coursetaking measures 
and their relations to measures of academic performance. Below is a brief list of the key analyses 
for the 11 participating TUDA jurisdictions that can be reported from this pilot study: 


e estimated mean credits earned by eighth-grade public school students between sixth and 
eighth grades; 


e overall grade point average earned by eighth-grade public school students on core academic 
courses (i.e., English, mathematics, science, and social studies courses) and other academic 
or nonacademic courses (i.e., fine arts, foreign languages, and career and technical education 
courses); 


e the proportion of the eighth-grade public school students who earned high school credits in 
courses between sixth and eighth grades; 


e the relationships between eighth-grade public school student coursetaking and performance 
on NAEP and state mathematics and reading assessments; and 


e eighth-grade public school students reporting of levels of persistence in learning, enjoyment 


in solving complex problems, and interest and enjoyment in mathematics and reading in 
relation to their grade point averages in mathematics and English courses. 
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Chapter 1. Study Planning and Design 
Introduction 


The National Assessment of Educational Progress (NAEP) 2017 pilot study of the Middle 
School Transcript Study (MSTS) is the first NAEP transcript study to focus on the middle school 
level. It is also the first NAEP transcript study to collect student coursetaking data at the Trial 
Urban District Assessment (TUDA) level and to rely solely on the electronic submission of 
course catalogs and student transcripts. 


Information about operational aspects of the pilot study, such as sampling and recruitment, data 
collection, coding, and linking to the NAEP 2017 assessment data, is provided in this report. 
Findings and recommendations are also included for future studies that conduct electronic data 
collection for course catalogs and student course records. 


The following sections describe the background and rationale of the study, including a 
description of the study design. 


Background and Rationale 


The NAEP 2017 MSTS pilot study was conducted by the National Center for Education 
Statistics (NCES), part of the U.S. Department of Education (OMB# 1850-0803 v.172 and 
v.180). Data for this pilot study were collected in conjunction with the NAEP 2017 eighth-grade 
mathematics and reading assessments. The data were linked to the NAEP assessment scores and 
the contextual questionnaire data so that relationships between student coursetaking and NAEP 
performance could be evaluated. 


The purpose of this study was to investigate the feasibility of the following activities: 


e collecting MSTS data that relied solely on the electronic submission of course catalogs 
and student course records; 


e linking coursetaking data with the NAEP assessment and contextual questionnaire data; 


e providing educators and education researchers with valuable information that would 
allow for new analyses and reporting in adolescent coursetaking and on issues that are 
relevant at the middle school level; and 


e examining the use of electronic student transcript and course records data submissions for 
NAEP transcript-based studies. 


Study Design 


Westat was contracted by NCES to carry out the recruitment, data collection, coding of course 
catalogs and student course records, analyses, and reporting for the NAEP 2017 MSTS pilot 
study. The Educational Testing Service (ETS) was contracted by NCES to prepare student NAEP 
assessment scores (known as plausible values) for analyses with data collected from this pilot 


study. Additional information about generating the plausible values for the NAEP 2017 MSTS 
pilot study is provided in Chapter 4 of this report. 


This pilot study proceeded in three major stages: recruitment, data collection and coding, and 
analysis and reporting. The study was limited to large urban districts, referred to as TUDA 
jurisdictions, to help evaluate and streamline operations for potential future NAEP TUDA-based 
middle or high school transcript studies, as well as other transcript studies that rely on electronic 
records. Participation in this study was strictly voluntary. TUDA jurisdictions that volunteered to 
participate agreed to submit information for all eighth-grade students enrolled in the schools 
sampled for the NAEP 2017 TUDA-based assessments. 


In the first stage of this pilot study, Westat prepared materials and solicited TUDA jurisdictions 
for participation. Westat also held webinars with the NAEP TUDA coordinators to ensure that 
the participants interested in volunteering understood their responsibilities and the requirements 
for the study and had the opportunity to ask questions about the pilot. 


In the second stage, data for the pilot study were collected and coded. The district information, 
course catalogs, student course records, and associated student education information were 
collected electronically. The district information and course catalogs were collected soon after a 
TUDA jurisdiction agreed to participate in the study, and the student course records and 
associated education information were collected during the following summer. Catalog courses 
were coded using the School Courses for the Exchange of Data (SCED) course classification 
system. Student course records were matched to the catalog course records through course ID 
numbers located on both the course catalog and student course records. 


In the third stage, the NAEP 2017 assessment scores of the students were adjusted using 
conditional variables and weights provided by Westat to ETS using secure protocols already 
developed for NAEP. Those variables and weights were used by ETS to generate scores (known 
as plausible values) for the students in this pilot study. Preliminary analyses focusing on 
coursetaking measures, relations to NAEP performances, and other education outcome variables 
were conducted. The methodological report on the MSTS pilot study focused on documenting 
the technical aspects of the study, as well as the analytic possibilities of linking MSTS data to 
NAEP assessment scores, NAEP contextual variables, and state assessment scores provided by 
the TUDA jurisdictions. In addition, each TUDA jurisdiction receives an individualized report 
based on the available data provided by the TUDA jurisdictions. The analyses in the TUDA- 
specific report include findings related to estimated credits earned, grade point averages, linking 
NAEP assessment scores and contextual variables, and state assessment scores, if data are 
available. Both the methodological report and the individual TUDA reports are not to be released 
to the public. 


Sampling, Recruitment, and Participation Requirements 


The primary sampling objective of this pilot study was to collect middle school student 
coursetaking information from a sample of the TUDA districts interested in participating in the 
study. The study initially aimed to seek voluntary participation from six to eight TUDA 
jurisdictions that were part of the NAEP 2017 TUDA-based assessments but was expanded to 
include all 11 TUDA jurisdictions that expressed interest. 


Recruitment materials included a description of benefits to TUDA districts of participating in the 
study (appendix A), a letter of solicitation (appendix B), and a brochure about the NAEP 2017 
MSTS pilot study (appendix C). These materials were sent in early December 2016 to the 27 
TUDA jurisdictions participating in the NAEP 2017 TUDA-based assessments, following the 
OMB approval of the pilot study. 


A follow-up web meeting with interested TUDA coordinators or representatives was conducted 
in December 2016 to further explain the objectives of the study and its data collection process 
and to field any questions the representatives had. Interested TUDA districts were asked to 
complete and submit a letter of interest to NCES before the end of January 2017 (appendix D). 


TUDA jurisdictions that were interested in participating in the NAEP 2017 MSTS pilot study 
needed to meet the following criteria: 


e anelectronic school records system and the ability to transmit student records 
electronically; 


e the ability to provide electronic course catalogs containing courses available to students 
from sixth through eighth grades; and 


e course ID numbers available on the electronic student records that match the course ID 
numbers available in the electronic course catalogs. 


For the purpose of this pilot study, electronic records systems were defined as school course 
listings and student course records stored in electronic database formats, such as Microsoft Excel 
or Access files, statistical software databases (e.g., SAS, SPSS, STATA), or file formats that can 
be easily converted into databases (e.g., comma-separated values or extensible markup language 
files). Portable document format (PDF) transcripts are considered analogous to paper copies 
because they require the NCES data collection contractors to manually enter information from 
the transcripts into a database. 


An electronic records system, which also can be referred to as an electronic-based student 
information system, is a centralized database (or set of databases) of student education 
information that maintains data on students’ educational progress from elementary to secondary 
schools. The system may be able to identify courses in which students are currently enrolled or 
have taken in the past, document grading or grading systems, provide results of student 
assessment scores, and track student attendance and other student-level data. 


Exhibit 1 shows yellow stars to represent the 11 TUDA jurisdictions that volunteered to 
participate in the pilot study and grey circles to represent nonparticipating TUDA jurisdictions. 


Exhibit 1. Participating TUDA jurisdictions in the NAEP 2017 MSTS Pilot Study 
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SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot study, 2017. 


All 11 TUDA jurisdictions that volunteered to participate in the pilot study met the study’s 
participation criteria and were able to complete the district information form and submit the 
course catalogs and the student course records electronically. 


Burden Estimate 


Because data was solely collected electronically at the district-level, the NAEP 2017 MSTS pilot 
study did not result in any known burden to the students, teachers, or schools. The level of 
burden was estimated only to be limited to the TUDA district administrative and information 
technology staffs to review, approve, and implement the request for the electronic submissions of 
course catalogs and student course records data, along with electronic supporting documentation. 
Table 1 lists the hourly estimates of response burden for each participating TUDA jurisdiction by 
activity (as approved in the OMB package). The hourly burden was estimated to be no more than 
12 hours for each participating TUDA jurisdiction for submitting the required data. 


Table 1. Estimated hourly response burden for each participating TUDA jurisdiction in 
the NAEP MSTS Pilot Study, by activity: 2017 


Activity Hours per response 
Interview to collect initial MSTS information 1 
Collection of District Information Form 2. 
Submission of course catalogs to secure FTP site 1 
Submission of student course records to secure FTP site 8 
Total 12 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


A follow-up email communication was sent in late April 2017 to the participating TUDA 
jurisdictions asking about the time it took to complete the district information form and submit 
the course catalogs. The 11 participating TUDA jurisdictions reported estimated burden hours 
are within the estimated hourly burden listed in Table 1. For the student course records and 
education-related data submission, the participating TUDA Jurisdictions noted the submitting 
process was straightforward and efficient once project staff at Westat provided additional details 
and answers to questions related to student course records. 


Chapter 2. Data Collection 


Overview 


Data collection for the NAEP 2017 MSTS pilot study occurred in three phases, first collecting 
district information forms, then course catalogs, and finally student course records with 
associated education information. Participating TUDA jurisdictions were responsible for 
transferring the electronic files to secure NAEP file transfer protocol (FTP) sites hosted at 
Westat. An MSTS Help Desk was provided by Westat in case TUDA representatives needed 
technical assistance preparing or submitting electronic files. 


No individual schools were involved in the data collection process. As mentioned previously, the 
data collection was done at the TUDA level. Therefore, there was no requirement for field staff 
to collect student course records at the participating NAEP assessment schools as has been 
traditionally done with the NAEP HSTS. Additionally, an automated approach was provided to 
collect and process the student course records data, with each participating TUDA jurisdiction 
electronically submitting and uploading the data files containing student coursetaking 
information to the secured FTP site at Westat. These procedures limited the need for a keying 
unit to enter the collected course information into analyzable databases, which were necessary 
for the previous paper-based student course records data collection operations. This reduced the 
time and cost needed to process the student course records, and eliminated data entry errors that 
could occur in student course records keying operations. 


Because the NAEP 2017 MSTS pilot study falls under the NAEP program, the collection of 
electronic student course records fell under the auspices of the overall NAEP data collection 
procedures and protocols (described in OMB #1850-0928 v. 1—2), and student course records 
disclosure is permitted under the exception of the Family Educational Rights and Privacy Act to 
the general consent requirement that permits disclosures to authorized representatives of the 
Secretary for the purpose of evaluating federally supported education programs (34 CFR §§ 
99.31 [a][3][iii] and 99.35). The following sections describe each of the data collection phases in 
more details. 


District Information Form 


The district information form asked for information related to the availability of electronic 
course catalogs, whether the course catalogs included course descriptions, the availability of 
course ID numbers in the course catalogs and on student course records, as well as various data 
elements of the student course records (see appendix E). These data elements are needed to code 
the course catalogs and student course records. Table 2 lists the major required and optional data 
elements listed in the district information form. 


Table 2. Major required and optional data elements listed in the NAEP MSTS Pilot 
Study’s district information form: 2017 


District-level education information Is data element required? 
Eighth-grade completion requirements Yes 
Credits earned for a year-long course Yes 
Common grading standard Yes 
State or district assessment scores Strongly recommended, but not required 
Districtwide education programs Strongly recommended, but not required 
School calendar (semester vs. quarter) No 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


The district information form also asked for the eighth-grade completion requirements, the 
number of credits earned for a year-long course, the grading standard, the availability of 
districtwide education programs, and the state or district assessments taken between sixth and 
eighth grades. Participating TUDA jurisdictions were able to complete and return the form 
between March and April 2017, which was in line with the target timeline for this phase of data 
collection. 


Course Catalogs 


District-level course catalogs were required to have, at a minimum, a list of school courses 
offered to students in each TUDA Jurisdiction between sixth and eighth grades. Each course was 
required to have a unique ID code that corresponded with the course ID codes found on the 
electronic student course records. TUDA jurisdictions were encouraged to include course 
descriptions and topics taught in each course. Course designations for special education, English 
language learner (ELL) courses, online courses, and courses where students earned high school 
credit were also collected. Table 3 lists the required and optional data elements for the district- 
level course catalogs. 


The collection of district-level course catalogs occurred shortly after the collection of the district 
information form. The majority of the participating TUDA jurisdictions were able to submit 
either course catalogs or course lists in April 2017, and all course catalogs were received by June 
2017. Overall, data collection for the district-level course catalogs was timely and well within the 
expected target timeline for this pilot study. 


Table 3. Required and optional data elements collected for the district-level course 
catalogs in the NAEP MSTS Pilot Study: 2017 


District-level course catalog information Is data element required? 
Course name Yes 
Course ID number Yes 
Course description Strongly recommended, but not required 
Number of credits earned for the courses Strongly recommended, but not required 
Designation for special education courses Strongly recommended, but not required 
Designation for English language learner courses Strongly recommended, but not required 
Designation for online courses Strongly recommended, but not required 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


Student Course Records and Education-Related Data 


The data collection of the student course records and associated education information started in 
June 2017. The student course records included courses taken by the students from sixth through 
eighth grades, with such data elements as course name, course ID number, grade level and year 
taken, course length, course credits earned, course grade earned, and designations for online and 
transfer courses. The associated education information included indicators of advancement to 
ninth grade, number of days absent, participation in academic-based programs, and state and 
district assessment scores for sixth through eighth grades. Tables 4 and 5 lists the required and 
optional data elements for the student course records and student education information. 


The majority of the participating TUDA jurisdictions were able to submit data for this stage 
during the targeted timeline of June through August of 2017. Some TUDA jurisdictions were 
delayed briefly because they had to wait for an official sign-off before they could release their 
education information data for the pilot study. 


Table 4. Required and optional data elements collected for the student course records in 
the NAEP MSTS Pilot Study: 2017 


Student course records information Is data element required? 
Course name Yes 
Course ID number Yes 
Grade level and year taken Yes 
Length of course Yes 
Number of credits Yes 
Grade earned (letter or numeric) Yes 
Level of courses Strongly recommended, but not required 
Designation for special education courses Strongly recommended, but not required 
Designation for English language learner courses Strongly recommended, but not required 
Designation for online courses Strongly recommended, but not required 
Designation for transfer courses Strongly recommended, but not required 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


Table 5. Required and optional data elements collected for the student education 
information records in the NAEP MSTS Pilot Study: 2017 


Student education information Is data element required? 
Eighth-grade completion status Yes 
Number of days absent Yes 
Student transfer status Yes 
Number of credits earned Yes 
Grade point average Yes 
State/district assessment scores Strongly recommended, but not required 


SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


Procedures for Data Quality Assurance in Data Collection 


The data quality staff checked each data file upon receipt for whether the required data elements 
were present. The NAEP 2017 MSTS pilot study adapted the standard quality assurance 
protocols developed for previous HSTS studies. The data quality staff specifically checked each 
file for whether course names, course ID numbers, grade levels and year taken, course credits, 
course grades, course lengths, and state assessment scores were provided on the student course 
records data files. The TUDA representatives were contacted if additional information or 
clarifications were needed. 


Overall, most participating TUDA jurisdictions did not provide course credits because they did 
not assign credits for middle school courses. For TUDA jurisdictions that provided course 
credits, the credits were converted to standardized Carnegie units of credit (i.e., one credit per 
year-long course). Carnegie unit of credit is defined as a single unit and is equal to 120 hours of 
classroom time over the course of a year. The converted Carnegie credits were compared to the 
original credits provided by the TUDA jurisdictions as an additional data quality assurance 
measure. For the TUDA jurisdictions that did not provide course credits, the course credits were 
derived using the Carnegie credit standard based on the course length data provided in the 
student course records files. 


For participating TUDA jurisdictions that provided numeric and letter grades earned in middle 
school courses, the grades were standardized to a four-point grading scale (A, B, C, D, and F) as 
a standard protocol developed for previous HSTS studies. Points were assigned to each letter 
grade (A = 4, B=3, C=2, D= 1, and F = 0) and multiplied by the number of Carnegie credits 
earned for the course. Dividing the sum of grade points by the sum of Carnegie credits earned 
resulted in a grade point average for those courses. Courses in which a student did not receive a 
letter grade, such as pass/fail and ungraded seminars, did not factor into the grade point average 
calculation. All courses were weighted equally in the calculation of grade point averages. This 
process did not standardize for differences in grading practices across the participating TUDA 
jurisdictions. 


Chapter 3. Course Coding Operation 


Course Coding Training 


The course coding operation began with a two-day training session conducted by experienced 
HSTS coding supervisors to ensure coding standardizations and high coding quality. The coding 
unit was trained on coding the district-level course catalogs and student course records using 
version 4.0 of the SCED classification system. This is the first time that SCED was used in a 
NAEP transcript-based study. The coders were trained specifically on how to use the SCED, and 
used practice course catalogs during training to become familiarized with the using the SCED. 


After completing the coding training, the coding unit began to code the district-level school 
course catalogs. Inter-rater reliability and quality control checks were regularly conducted to 
ensure high coding and data quality. Coding of the course catalogs was performed as the course 
catalogs and course lists were being received. 


Before the student course records were collected, a one-day training session was held to instruct 
coders on the automated and manual matching process of student course records. The course ID 
numbers provided on both the course catalogs and student course records served as the link that 
automated the matching process. If course ID numbers were not available on the course catalogs, 
coders manually matched course titles from student records to the course titles in the catalogs. 


The coding unit began the automated matching process as soon as student course records were 
received. At this time coding supervisors also reviewed all manually-coded student course 
records for accuracy by cross-checking coded values with the course catalogs and student course 
records. 


Adapting School Courses for the Exchange of Data (SCED) 


The NAEP 2017 MSTS pilot study used the SCED classification system to code the course 
catalogs and student course records. The SCED is a new course classification system that was 
developed by the National Forum on Education Statistics for prior-to-secondary and secondary 
school courses. It is maintained by a working group of state and local education agencies and 
hosted by NCES. Different from the Classification of Secondary School Courses (CSSC), which 
was the previous course classification system, the SCED system consists of four elements that 
provide a basic structure for classifying course content. These elements are a five-digit course 
code, course level (i.e., basic/remedial, general/regular, enriched/advanced, and honors), grade 
span, and course sequence. One of the major changes between CSSC and SCED is that SCED 
does not include a specific category for special education courses. Thus, specific to the NAEP 
2017 MSTS pilot study, additional attributes were added to the SCED codes to capture course 
designations for special education, as well as ELL courses, online courses, and courses where 
high school credits were earned. 
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Data Quality Assurance Protocol for Coding Operation 


The coding of course catalogs and student course records was conducted by coders who were 
trained by experienced HSTS supervisors. Each course catalog was double-coded for coding 
reliability. An intercoder reliability rating of 95 percent was maintained throughout the coding 
operations, which lasted between June and August 2017 for course catalogs and September 2017 
for student course records. Additionally, between 10 and 15 percent of the courses were 
randomly selected for additional coding verification by the coding supervisor to ensure 
consistency as part of the data quality assurance protocol. 


Because of the requirement to have course ID numbers available on both the course catalogs and 
student course records, over 90 percent of the student course records were electronically matched 
to the course catalogs by Westat’s proprietary course coding software. This key data element 
made the coding of the student course records significantly faster to complete than coding the 
student course records individually without the matching of course ID numbers. Overall, the 
coding operation was completed in three months for course catalogs, which included 
approximately 6,000 course titles, and in only one month for about 123,800 student course 
records. 


During the coding operation, technical discussions were held regularly to resolve and document 
any ambiguities or disagreements between course codes. Systematic programming checks were 

regularly performed to ensure consistency and data quality (e.g., SCED codes assigned to match 
the description of the course titles). 


After coding of the course catalogs and student course records was completed, additional 
systematic programming checks were performed for consistency, and preliminary frequency 
analyses were conducted to identify similar course patterns within and across the participating 
TUDA jurisdictions. In addition, TUDA jurisdictions were contacted to confirm and verify the 
results of coding on course catalogs. 


Inclusion Criteria for Coursetaking Analysis 


Before student course records data could be used in the analysis, a set of inclusion criteria was 
applied. Adapted from previous HSTS data quality inclusion criteria, each student must have 
course records including: 


e at least two years of data, including eighth-grade courses taken during the 2016-2017 
school year; 


e atleast 12 Carnegie credits in courses listed; and 
e Carnegie credits earned in English courses. 


The percentage difference between collected and analyzed student course records was similar to 
the previous HSTS studies after inclusion criteria are applied. The criteria for inclusion in the 
analyses were established to ensure that the course records for each student were complete and 
valid. 
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Chapter 4. Constructing Databases and Linking to the NAEP 2017 Data 


Constructing Databases 


One of the major goals for the NAEP 2017 MSTS pilot study is to provide coursetaking data that 
supplements the NAEP assessment and contextual questionnaire data at the middle school level. 
To do that, student course records collected for the pilot study needed to be linked to the NAEP 
2017 data for the students who are in the NAEP 2017 mathematics and reading assessment 
samples. Because NCES requires that the TUDA jurisdictions not know which of their students 
were sampled for the NAEP assessments, procedures were put in place to inform the TUDA 
jurisdictions which students they need to submit student course records without revealing the 
student sample. As mentioned previously, TUDA jurisdictions who volunteered to participate in 
NAEP 2017 MSTS pilot study agreed to submit information for all eighth-grade students 
enrolled in the schools sampled for the NAEP 2017 TUDA-based assessments. Confidential 
information is protected during transmission to and from NAEP systems by the use of robust 
secure file transfer protocol (FTP) platforms and of data encryption technologies, such as secure 
sockets layer, secure shell, and digital certificates and signatures that encrypt data, validate data 
integrity, and authenticate the parties in a transaction. 


As part of the NAEP 2017 assessments, each sampled school submitted a list of all eligible 
eighth-grade students. The list included a NAEP-assigned unique ID for each student. The 
unique ID served as the linking variable between the students sampled for NAEP assessments 
and the pilot study to maintain the privacy of student and school identities. Specifically, when 
the electronic student course records were provided by district personnel, they completed the 
following process by downloading a list of all students included on the roster of students, plus 
any new enrollees added during the Update Student List process (this includes the information 
previously provided by the district); adding student course record information to the downloaded 
file; and uploading the student course records file to the secure FTP site. 


Westat MSTS project staff then added the student course records data to the student demographic 
records by matching on the NAEP-assigned unique student identifier. This student course 
records information was then sent to Educational Testing Service (ETS), who matched the 
information with the data from NAEP assessment by using the NAEP-assigned unique student 
identifier. At no time was any personally identifiable information about the students transmitted 
with the student course records sent to ETS, nor merged with the NAEP assessment data. 


Creating Student Weights 


Four sets of student weights were created for the NAEP 2017 MSTS pilot study, similar to the 
previous NAEP HSTS studies. Each set of weights included a survey weight, used to produce 
point estimates, and replicate weights, used to compute variances for point estimates. All four 
sets of weights were designed to produce estimates for all eighth-grade students in each 
participating TUDA jurisdiction. The type of weight that was used depended on the type of data 
the user was analyzing. 
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The first set of weights reflect the probability-sampling scheme used to arrive at the sample of 
students for whom a valid transcript was collected. These weights were constructed without 
regard to the NAEP participation. 


The second, third, and fourth sets of weights were linked to the eighth-grade NAEP assessments. 
The second set of weights were derived for the subsample of MSTS students for whom a valid 
transcript was collected and also took the mathematics assessment. Similarly, the third set of 
weights were derived for the subsample of MSTS students with valid transcripts and took the 
reading assessment. These weights allowed users to analyze the relationship between students’ 
mathematics or reading proficiencies, respectively, as measured by their NAEP assessment 
outcomes and students’ coursetaking in their middle school careers. 


The fourth set of weights were derived for the MSTS students with valid transcripts and took 
either the mathematics or reading assessments. This set of weights was used for NAEP 
contextual items that are asked of students on both assessments. 


Generating Plausible Values 


Before the student course records data could be analyzed, the NAEP mathematics and reading 
assessment scores for the sampled students needed to be adjusted to account for the additional 
student coursetaking information. A set of conditioning variables was prepared for ETS to 
generate the NAEP 2017 mathematics and reading plausible values for the eighth-grade students 
in the NAEP 2017 MSTS pilot study sample. The conditioning variables included most of the 
variables that were used in the analyses, such as various credits earned and grade point average 
variables. All four sets of student weights for the NAEP 2017 MSTS pilot study were also 
provided to ETS to generate the NAEP 2017 mathematics and reading plausible values for the 
pilot study. Because additional conditioning variables were used for the MSTS, the NAEP 2017 
eighth-grade NAEP mathematics and reading plausible values for the MSTS pilot study may 
differ from the plausible values generated for the TUDAs in the main NAEP study. Data files 
provided to and received from ETS used secured protocols established and implemented for the 
main NAEP operation. 
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Chapter 5. Findings of the Electronic Submission Feasibility Study 


All participating TUDA jurisdictions were able to submit required data files electronically and 
provide course IDs on both the course catalogs and student course records. Specifics of the 
electronic submission feasibility are described in the following sections. 


Types of Course Catalogs 


Seven out of 11 TUDA jurisdictions were able to offer course catalogs with course descriptions 
or web links to where course descriptions were available. Course descriptions offer additional 
information that helped to assign precise SCED codes, mainly if a course title includes 
information about different subject areas (i.e., History of Science and Technology). Other TUDA 
jurisdictions provided course lists with course titles, and coding of the courses relied solely on 
the course title information. Where necessary, the coding unit searched state and district websites 
to find additional information about the middle school curriculum and course offerings. Over 
6,000 course titles were collected and coded for this pilot study. 


Characteristics of Student Course Records 


Overall, approximately 123,800 sets of student course records (i.e., middle school transcripts) 
were collected, and 107,900 sets were analyzed from 11 TUDA jurisdictions for the NAEP 2017 
MSTS pilot study. Table 6 shows the approximated number of student transcripts collected for 
other NCES transcript studies between 1982 and 2013. 
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Table 6. Number of transcripts collected for selected NCES high school transcript 
studies: 1982-2013 


Approximate number 


NCES transcript study of transcripts' 
1982 High School and Beyond 12,700 
1987 NAEP High School Transcript Study 34,100 
1990 NAEP High School Transcript Study 21,500 
National Education Longitudinal Study of 1988 Second Follow-Up (1992) 17,300 
1994 NAEP High School Transcript Study 25,500 
1998 NAEP High School Transcript Study 25,000 
2000 NAEP High School Transcript Study 21,000 
Education Longitudinal Study of 2002 First Follow-Up (2004) 16,400 
2005 NAEP High School Transcript Study 27,200 
Education Longitudinal Study of 2002 Second Follow-Up (2006) 14,900 
2009 NAEP High School Transcript Study 37,700 
High School Longitudinal Study of 2009 (2013 Update) 25,200 


'Tncludes transcripts that were not included in the final reports because they were out of scope. Out-of-scope 
transcripts include transcripts from students who did not graduate in the year of the study, nonstandard transcripts 
that could not be incorporated or standardized with the other transcripts, and transcripts with less than three years of 
data. Each of the studies listed had transcripts that were out of scope. 

? Includes high school transcripts that were eligible but not responded in the 2013 update. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
High School and Beyond (HS&B), 1982; National Education Longitudinal Study of 1988 (NELS:88/92), Second 
Follow-Up, 1992; Education Longitudinal Study of 2002 (ELS:2002/04), First Follow-Up, 2004; Education 
Longitudinal Study of 2002 (ELS: 2002/06), Second Follow-Up, 2006; High School Transcript Study (HSTS), 
selected years, 1987-2009; High School Longitudinal Study of 2009 (HSLS:09), 2013 Update, High School 
Transcript Study. 


In comparison, the NAEP 2017 MSTS pilot study collected and analyzed nearly three times 
more student course transcripts than the NAEP 2009 HSTS, which was the largest transcript- 
based study between 1987 and 2013. This result provides evidence of the effectiveness of this 
pilot study regarding the volume of data collected electronically. The differences in the numbers 
of collected versus analyzed records were similar between this pilot study and previous HSTS 
administrations. 


Key Data Elements Provided on Student Course Records 


To help illustrate the ability of TUDA jurisdictions to provide key data elements for student 
coursetaking information, Table 7 shows the number of TUDA jurisdictions that were able to 
provide the major aspects of coursetaking measures. The major aspects of the coursetaking 
measures needed for the MSTS analyses were course credit, course grade, course length, and 
state assessment scores. Overall, three TUDA jurisdictions provided course credits on their 
student course records data files. The majority of the TUDA jurisdictions did not provide course 
credits because credits were not assigned for middle school courses. For these TUDA 
jurisdictions, the course credits were derived using the Carnegie credit standard (i.e., one credit 
per year-long course) based on the course length data provided in the student course records 
files. Almost all TUDA jurisdictions provided course grades and state assessment scores. 
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Table 7. Number of TUDA jurisdictions in the NAEP MSTS Pilot Study providing major 
aspects of coursetaking measures: 2017 


Coursetaking measure Provided Not provided/derived' 
Course credit 3 8 
Course grade 9 2 
Course length 11 0 
State assessment scores 10 1 


' Course credits were derived using the Carnegie credit standard (i.e., one credit per year-long course) based on the 
course length data provided in the student course records. 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 
Middle School Transcript Study (MSTS) Pilot Study, 2017. 


Summary of Key Analytic Results 


In assessing the feasibility of collecting and linking student coursetaking data with NAEP 
assessment data, several analytic inquiries were conducted to investigate the following questions: 


e whether the standard coursetaking measures (i.e., credits earned and grade point 
averages) could be adapted from the NAEP HSTS definitions and used in various course- 
related analyses; 


e how coursetaking measures relate to student performance on NAEP assessments; 
e how NAEP assessment scores relate to student achievement on state assessments; and 


e how coursetaking measures relate to NAEP student contextual indices. 


Differences among students by demographics (i.e., gender and race/ethnicity) and course-related 
variables were analyzed. Additional analyses on various academic programs were conducted to 
further examine the coursetaking measures and their relations to various measures of academic 
performance. Other exploratory analyses included identifying students at the middle school level 
who may be at risk of dropping out of high school and determining student access to various 
digital resources for learning in school and at home. Per the OMB# 1850-0803 clearance (v.180), 
the study findings do not constitute official statistics and are not to be released to the public, but 
aggregate results were provided to the participating TUDA jurisdictions for feedback. Below is a 
brief list of the major findings that can be reported from this pilot study: 


e estimated mean credits earned by eighth-grade public school students between sixth and 
eighth grades for the 11 participating TUDA jurisdictions; 


e overall grade point average earned by eighth-grade public school students on core academic 
courses (i.e., English, mathematics, science, and social studies courses) and other academic 
or nonacademic courses (i.e., fine arts, foreign languages, and career and technical education 
courses) for the 11 participating TUDA jurisdictions; 
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the proportion of the eighth-grade public school students who earned high school credits in 
courses between sixth and eighth grades; 


the statistical relationships between eighth-grade public school student coursetaking and 
performance on NAEP and state mathematics and reading assessments (e.g., group 
comparisons and correlations); and 


eighth-grade public school students reporting of levels of persistence in learning, enjoyment 


in solving complex problems, and interest and enjoyment in mathematics and reading in 
relation to their grade point averages in mathematics and English courses. 


17 


Chapter 6. Conclusions and Recommendations 


Conclusions 


Findings from the NAEP 2017 MSTS pilot study showed that all 11 participating TUDA 
jurisdictions could submit required MSTS data files electronically. TUDA jurisdictions were able 
to provide course IDs on both the course catalogs and student course records. Having the course 
IDs on both data files made coding the student course records more efficient. The majority of the 
TUDA Jurisdictions were able to offer course catalogs with course descriptions or web links to 
where course descriptions were available. Course descriptions offer valuable information that 
helped to assign accurate SCED codes, especially in cases where a course title included 
information about different subject areas (e.g., History of Science and Technology). 


The study also demonstrated the feasibility of linking student coursetaking information with 
NAEP assessment data. Preliminary data analyses focused on the student coursetaking measures, 
and how those measures relate to the NAEP assessment score and contextual data, state 
assessment scores, and participation in special academic programs. These analyses revealed 
potential similarities as well as variabilities in the results among the TUDA jurisdictions. 
Preliminary data analyses that focused on student coursetaking measures (i.e., credits earned and 
grade point averages) and NAEP assessment scores demonstrated similar general patterns in 
these relationships that have been illustrated in the previous HSTS studies. Additionally, 
preliminary analyses that focused on the relationships between the NAEP assessment scores and 
students’ performance on their state assessments provided additional approaches to examine 
student achievement from different sources. 


Further analyses were conducted to examine the relationships between the new NAEP 2017 
contextual indices on affective disposition and the coursetaking measures, such as how grade 
point averages relate to student reporting of levels of persistence in learning, enjoyment in 
solving complex problems, and interest and enjoyment in mathematics and reading. Results from 
these analyses on the contextual indices provide additional evidence for the construct validity of 
these indices. 


These results show that data collected from this pilot study can supplement the NAEP 
assessment and contextual questionnaire data to allow richer analyses at the middle school level. 
The pilot study also shows that policy-related analyses can be conducted at the middle school 
level and may help identify additional analyses for future middle school transcript studies. 


Furthermore, results from this study can shed light on the fundamental core and supplementary 
courses that are being taught in middle school, and the degree of challenge in coursetaking 
patterns that may promote and facilitate intellectual growth. Concerning the student dropout 
issue, the signs that a student is at risk of dropping out of school tend to first appear in middle 
school. Student coursetaking data from middle school years can help to identify specific 
maladaptive coursetaking patterns that may be indicative of a high likelihood of students 
dropping out. Conversely, it can also help to identify adaptive academic coursetaking behaviors 
and possible academic success pathways that help keep students in school. 
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Recommendations 


Based on the operational experiences and findings from this pilot study, the recommendations 
involving the use of curriculum-related terminology in middle school, adapting the SCED 
classification system for middle school courses, and additional data quality assurance 
requirement are described in the following sections. 


The Use of Terminology in Middle School 


Experiences from the various stages of the operations and logistics for this pilot study indicate 
that terminology and education policies related to coursework, coursetaking, and curriculum in 
middle school are different from high school. For example, the term “student transcript” is often 
not used at the middle school level, where the courses students took are instead referred to as 
“student course records.” The meaning and definition of grade systems varied at the middle 
school level as well. For example, the notion of eighth-grade completion requirements was 
deemed more ambiguous than high school graduation requirements. Almost all of the TUDA 
jurisdictions in the NAEP 2017 MSTS pilot study were not able to provide a complete list of 
curriculum requirements that students must achieve before the end of eighth grade. Future 
studies should ask districts to provide specific course requirements that students need to attain 
for promotion to next grade level, and information on how students are promoted to high school 
(e.g., passing specific courses and state assessments). 


Suggestions to the SCED Working Group 


Courses at the middle school level tend to be more integrated with each other and may not fit 
into one particular SCED category (e.g., 3-D Design and Printing as an art technology course, 
Foundations of Robotics as a history of robotics course, and Movement and Technology as a 
dance course with various technologies used). The coding unit also encountered challenges in 
using the SCED classification system to code the course level for special education courses. 
Exhibit 2 provides a list of major issues documenting the challenges in using the SCED 
classification system for coding courses at the middle school level as well as recommendations 
for future studies. 
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Exhibit 2. 


school courses in the NAEP 2017 MSTS Pilot Study 


Major issues identified with using the SCED classification system for middle 


Course name _ Issue SCED code decision Recommendations 

Project Lead the Way: PLTW program specifically | 21099 Suggestion to add a code for 

Gateway (PLTW Gateway) designed for students in (Technology, Other) PLTW Middle School to 
middle school. The SCED cover all courses under 
currently does not have PLTW-Gateway units. 
specific codes for these 
courses. 

Cambridge Program An accelerated program 01099, 02099, 03099, with H Suggestion to add codes for 


for Course Level the Cambridge program for 
the main academic subjects 
(i.e., English/language arts, 


mathematics, and science). 


designed for students in 
middle school that focuses 
on improving skills in 
mathematics, reading, 
science, and English. The 
SCED currently does not 
have specific codes for these 
courses. 


IB Middle Years Programme The SCED currently does 01099, 02099, 03099, etc., Suggestion to add additional 
(IB MYP) not have specific codes for with H for Course Level codes for the new IB MYP 
new IB MYP courses courses or add “Other IB 
MYP” codes to existing IB 
MYP subject areas. 
Braille No code in the SCED 22999 Additional discussion may 
(MISCELLANEOUS— be needed for courses 
OTHER) offered to students who are 


visually impaired. 
Additional discussions are 
needed. 


No code in the SCED Coded to regular courses. 
Flagged with SPED. Coding 
of course level depended on 
course descriptions. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, 


Middle School Transcript Study (MSTS) Pilot Study, 2017. 


Special Education 


For most of the participating TUDA jurisdictions, neither course titles nor course descriptions 
clarified whether the special education courses were general content courses modified for special 
education students (i.e., a general/regular course level) or courses that taught content below the 
grade level (i.e., a basic/remedial course level). 


Additional Data Quality Assurance Requirement 


Because the NAEP 2017 MSTS pilot study was the first NAEP transcript study to focus on the 
middle school level, and because it relied solely on the electronic transmission of course catalogs 
and student course records, districts were contacted several times throughout this study to review, 
verify, and confirm the course coding as additional quality assurance measures. It is duly essential 
to request the TUDA representatives to provide a final review and approval of their results before 
the release of the data. 
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Appendix A—Recruitment Communication: 
Benefits to TUDA Districts Participating in the Study 


Recruitment Communication: 
Benefits to TUDA Districts Participating in the Study 


1. Customized TUDA-level data summaries at no cost to TUDAs. Each participating TUDA will 
receive a data summary containing analyses specific to issues important to their district, 
deepening their understanding of topics such as students’ course trajectories, curriculum 
rigor, and achievement gap problems. MSTS staff will work with each TUDA to customize 
these data summaries at no cost to the districts. Note that given this is a feasibility study, 
the summaries will be unofficial and cannot be published or shared outside of the district 
office. 


2. Expanded TUDA-level NAEP analysis with the MSTS feasibility study data. Data collected 
from the MSTS feasibility study will provide TUDAs more contextual data to explain their 
eighth-grade students’ NAEP assessment scores. What courses they took and their 
performance in those courses, combined with their attitudes toward the courses as asked in 
the student questionnaire, will provide a more complete picture of how students perform 
on the eighth-grade NAEP assessments. 


3. New measures to focus on key issues in middle school at TUDA-level. Measures of middle 
school curriculum levels will be developed to look at the rigor of middle school curriculum 
and examine academic tracking issues. The MSTS feasibility study will also be able to take a 
comprehensive look at the digital divide issue at the middle school level by combining data 
from the NAEP questionnaires and the information about online courses provided by the 
TUDAs. By examining student coursetaking patterns, maladaptive coursetaking behaviors, 
and other factors that are typically associated with student dropouts (i.e., as absenteeism, 
and grade level/course repeats), middle school success indicators could be developed to 
further identify students who are at risk of dropping out. 


4. No burden to schools and students. Unlike previous transcript studies where data 
collections are done at the school level, data collection for the Middle School Transcript 
Study (MSTS) 2017 feasibility study will be done at the district level, thereby placing no 
burden on schools and students. 


5. Minimal labor cost to the districts for data collection. Because the MSTS 2017 feasibility 
study will be a part of the NAEP 2017 Grade 8 assessments, there will be no cost to the 
districts, except for minimal labor cost from TUDA data personnel to extract and prepare 
electronic data files to submit for this study. 


A-2 


Appendix B—Recruitment Letter From NCES 


B-1 


Recruitment Letter From NCES 
Dear TUDA District Superintendent (each letter to be individually addressed): 


| am writing to invite your district to participate in a new initiative that will expand the National 
Assessment of Educational Progress (NAEP) to include a transcript study focusing on middle school 
grades. In 2017, the National Center for Education Statistics (NCES) will conduct a feasibility study of the 
collection of middle school student coursetaking information for students selected for the NAEP grade 8 
assessments. This feasibility study will rely solely on electronic submission of course catalogs and 
student transcripts and will provide educators and policymakers a new resource for exploring the 
relationship between students’ coursetaking patterns and their performance on NAEP. 


The feasibility study of the groundbreaking Middle School Transcript Study (MSTS) is the first NAEP 
transcript study to focus on middle school grades. While it may eventually be expanded to schools 
across the country, your district and other large urban districts are the first to have an opportunity to 
participate. This is also the first transcript study to rely entirely on electronic submissions, which will 
substantially reduce cost, time, and effort over previous transcript studies. In return, you will receive 
detailed district-level analyses. 


Costs for this feasibility study will be the responsibility of the federal government, as with NAEP 
assessments at grades 4 and 8. NCES contractors will work with your district staff to obtain electronic 
course catalogs and student transcripts; this is the only additional burden placed on your district. There 
is no additional burden at the school level. The MSTS is part of an overall movement in NAEP transcript 
studies toward relying on more efficient electronic records. 


To meet critical deadlines for data collection preparation, it is necessary to identify interested districts 
now. In the event that NCES receives letters of interest from more districts than funding can 
accommodate, we will select a sample of districts that is as regionally representative and 
demographically diverse as possible, while also considering the quality of electronic records in each 
district. 


NCES will host a WebEx on [DATE] for jurisdictions interested in participating in the study to review the 
details of the study, including the study timeline, data to be collected as part of the transcripts, and 
reporting plans. Details about the WebEx will be emailed to your NAEP TUDA Coordinator. 


If your district wishes to participate in this innovative feasibility study, please complete and submit the 
enclosed form by close of business on [DATE]. Districts will be notified of their participation by [DATE]. 


Thank you for considering participation in this important activity. NCES will keep you apprised of the 
status of the feasibility study. If you have any questions, please contact Linda Hamilton at 202-245-6360 
or linda. hamilton@ed.gov. 


Sincerely, 

Peggy G. Carr, Ph.D. 

Acting Commissioner 

National Center for Education Statistics 


cc: Mike Casserly, Council of the Great City Schools (CGCS) 
TUDA Coordinator (edit with TUDA coordinator name) 
Enclosures: Benefits to TUDAs [Appendix A], Brochure [Appendix C], Letter of Interest [Appendix D] 
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TUDA Recruitment Brochure Text 


COVER 
NAEP 2017 Middle School Transcript Study (MSTS) Feasibility Study 


INTERIOR 

What is the Feasibility Study of the NAEP 2017 Middle School Transcript Study? 

As part of the National Assessment of Educational Progress (NAEP) 2017 grade 8 assessments, the 
feasibility study of the NAEP 2017 Middle School Transcript Study (MSTS) is the first landmark NAEP 
study to examine the feasibility of linking middle school students’ coursework from sixth through eighth 
grades with NAEP 2017 grade 8 assessment data. 


The NAEP 2017 MSTS feasibility study is also the first NAEP transcript study to provide student 
coursetaking data at the district level and to rely solely on the electronic submission of course catalogs 
and student transcripts. 


The NAEP 2017 MSTS feasibility study is conducted by the National Center for Education Statistics 
(NCES), part of the U.S. Department of Education. The data collection will be a part of the NAEP 2017 
assessments for mathematics, reading, and writing. 


Why is NAEP 2017 MSTS feasibility study important? 

The NAEP 2017 MSTS feasibility study provides valuable student coursetaking data that supplements 
NAEP assessment data along with school, teacher, and student questionnaire responses. It allows for the 
comparison of student coursetaking data within jurisdictions and informs the decision if additional 
Middle School Transcript Studies should be conducted in the future. In addition, the MSTS feasibility 
study relies on electronic school records for collecting data for this and future transcript studies, which 
eliminates the burden on individual schools to provide paper copies of transcripts. 


Why participate in NAEP 2017 MSTS feasibility study? 

The study promotes a deeper understanding of student academic achievement by examining 
coursetaking patterns, middle school curriculum rigor, and potential student academic tracking. Coupled 
with the NAEP school, teacher, and student survey questionnaire information, this study provides 
valuable information to help identify academic coursetaking behaviors and possible academic pathways 
to success. 


Study Timeline: 

Winter 2017 Participation Recruitment 
Spring/Summer 2017 Data Collection 
Spring/Summer 2018 Reporting 


Requirements for Participation in the NAEP 2017 MSTS Feasibility Study 
Districts interested in participating in the NAEP 2017 MSTS feasibility study must have the following: 
e Anelectronic school records system and the ability to transmit student records electronically, 
e The ability to provide electronic course catalogs containing courses available to students from 
grades 6 through 8, and 
e The course ID number available on the electronic student record serving as a linking variable to 
match course numbers in the course catalogs. 


BACK 

The National Assessment of Educational Progress (NAEP) is a congressionally mandated project 
conducted by the National Center for Education Statistics (NCES), within the U.S. Department of 
Education and the Institute of Education Sciences. 


NCES is the primary federal entity for collecting, analyzing, and reporting data related to education in 
the United States. It fulfills a congressional mandate to collect, collate, analyze, and report full and 
complete statistics on the condition of education in the United States; conduct and publish reports and 
specialized analyses of the meaning and significance of such statistics; and assist state and local 
education agencies in improving their statistical systems. 


NCES activities are designed to address high-priority education data needs; provide consistent, reliable, 
complete, and accurate indicators of education status and trends; and report timely, useful, and high- 
quality data to the U.S. Department of Education, the Congress, the states, other education 


policymakers, practitioners, data users, and the general public. 


For questions about the NAEP 2017 MSTS feasibility study, please email MSTS@westat.com. 
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Appendix D—Recruitment Material: Letter of Interest From Districts 


Letter of interest from districts 


[Insert Date] 


Linda Hamilton 

National Center for Education Statistics 
Potomac Center Plaza 

550 12th Street SW 

Room 4093 

Washington, D.C. 20202, USA 


Dear Ms. Hamilton, 


(insert TUDA District’s name) is interested in participating 
in the feasibility study of the NAEP 2017 Middle School Transcript Study (MSTS). | understand that, if 
NCES receives letters of interest from more TUDA districts than funding can accommodate, they will 
select a sample of districts that is as regionally representative and as demographically diverse as is 
possible; the quality of electronic school records may also be taken into consideration. Thank you very 
much. 


Sincerely, 


(signature) 


Please mail to: 


Linda Hamilton 

National Center for Education Statistics 
Potomac Center Plaza 

550 12th Street, SW 

Room 4093 

Washington, D.C. 20202, USA 
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Appendix E—District Information Form 
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DISTRICT INFORMATION FORM 


Course Catalog and Student Transcripts Fillable Questionnaire 


Thank you for agreeing to participate in the feasibility study of the 2017 NAEP Middle School 
Transcript Study. It is the first NAEP middle school transcript study, and it relies solely on the 
electronic submission of both course catalogs and student transcripts. The purpose of this form is to 
collect information about the middle school course catalog and student transcripts. Completing this 
form should take about 45 minutes, and you may need access to the district’s course catalog and 
student transcripts to answer the questions. 


All obtained student transcript data may be used only for research purposes and will not be disclosed 
or used in identifiable form for any other purpose except as required by law [Education Sciences 
Reform Act of 2002, 20 U.S.C. §9573]. 


1. Is there a publicly available online version of your district’s middle school course catalog; that is, a 
catalog of courses that are available to students in the sixth through eighth grade? 
Yes (continue with Question 1a) o No (skip to Question 1b) 


a. If an online catalog is available, please list the web address where we can find this catalog, then 
skip to Question 2. 


b. If the catalog is not publicly available online, can you provide us with an electronic copy of the 
middle school course catalog? 
Yes O No 


If No, please explain below how we can get a list of middle school courses offered in your 
district, then skip to Question 4. 


2. Does the course catalog provide descriptions of the content for the courses (i.e., what topics are 
taught in the courses) or simply list the available courses? 


Yes, the catalog includes course content descriptions. 
No, the catalog only lists available courses. 


3. For each middle school course, does the course catalog provide the following information: 


a. the number of credits a student earns for passing the course? 
Yes O No 
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b. a designation in either the course title or course description that indicates if the course is a 
special education course? 
Yes Oo No 


c. a designation in either the course title or course description that indicates if the course is an 
English language learner course? 
Yes Oo No 


d. adesignation in either the course title or course description that indicates if the course is 
only available as an online course? 
Yes Oo No 


e. a designation in either the course title or course description that indicates if the student 
earns high school credit for passing the course? 
Yes O No 


4. Does the catalog (or whatever course lists can be provided to us) include an identification number 
for each course? 


Yes (continue with Question 4a) o No (skip to Question 4b) 


a. If you answered Yes to Question 4, are these identification numbers included (or can be 
included) on the electronic transcripts so that courses from the transcripts can easily be linked 
to the catalog or course lists? 

Yes Oo No 


b. If you answered No to Question 4, please explain below how you match courses between the 
catalog or course lists and the electronic transcripts. 


Next, we will ask some questions about the information that can be found on the electronic student 
transcripts. 


5. Is the following course information identified on the transcripts? If the information can be found on 
the transcripts, how are they indicated (e.g., a field on the record or notation in the course title)? 


a. Course level (i.e., regular, honors) o Yes co No 


b. Course taken off campus o Yes co No 


c. Special education course o Yes co No 
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d. English language learner course o Yes co No 


e. High school credit earned for the course o Yes o No 


f. Course taken online o Yes co No 


g. Transfer course o Yes co No 


6. Are the course titles listed on the transcripts the same or abbreviated versions of the course titles 
listed in the middle school course catalog? 
Yes O No 


If you answered No to Question 6, please explain below what differences there are between the 
catalog course titles and transcript course titles. 


Next, we would like to know how to interpret data found on the electronic transcripts. 


7. What graduation requirements must a student meet to advance from eighth grade to ninth grade? 


8. How many credits does an eighth-grade student earn for a year-long course; that is, a course taken 
for asingle period over the 2016-2017 school year or its block equivalent? 


Number of credits, 2016-2017: 


a. Is the number of credits earned for a year-long course different for either sixth or seventh 
grade? 
Yes O No 
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If you answered Yes to Question 8a, how many credits does a student in the following 
grades earn for a year-long course? 


Number of credits, sixth grade: 
Number of credits, seventh grade: 


b. Has this number of credits earned for a year-long course by sixth- through eighth-graders 
changed during the previous two school years? 
Yes O No 


If you answered Yes to Question 8b, how many credits did a student earn for a year-long 
course in the following school years? 


Number of credits, 2014-2015: 
Number of credits, 2015-2016: 


9. What grading system is used in the district for the eighth grade? (Mark one box.) 


Letter grade (A, B, C, D, etc.) 

Letter grade with modifiers (A, A-, B+, B, etc.) 

Pass/Fail 

Excellent/Satisfactory/Needs Improvement/Unsatisfactory 
Other (Please specify) 


a. Is the grading system different for either sixth or seventh grade? 
Yes O No 


If a different grading system is used in sixth and/or seventh grades, please indicate the 
grading system below. 


If a letter grading system (with or without modifiers) is used in the district for sixth, seventh, or 
eighth grades, please answer Question 10. Otherwise, skip to Question 11. 


10. What is the numerical range (on a 0 to 100 scale) for each of the letter grades used in the 
state/district? 
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Letter Grade or Alternate Range (or description, if range not possible) 
Symbol 
A+ 
A 


Pass 

Fail 

Excellent 
Satisfactory 

Needs improvement 
Unsatisfactory 


11. Are the credits earned and grading system the same for all courses, regardless of course level (i.e., 
special education, honors)? 
Yes O No 


If you answered No to Question 11, please explain the differences below (for example, honors 
courses earn more credits, special education courses use a different grading scale). 


12. Does the state or district require students to complete one or more assessments to advance from 
sixth, seventh, or eighth grade? 
Yes O No 


If you answered Yes to Question 12, please list these assessments and at what grade levels the 
students take them. 


13. Does the district offer any educational programs to students in sixth through eighth grades that 
focus on their coursework (i.e., offers them courses not normally taught to middle grade students, 
offers courses that supplement what is normally taught)? 

Yes Oo No 


E-6 


If you answered Yes to Question 13, please list these educational programs and at what grade levels 
the students take them. 


Next, we will ask you about some additional student data that we would like to collect at the same 
time we collect the student transcripts. These data are all related to the student’s academic standing 
and coursework in grades 6 through 8. 


14. Can the following student information be collected and sent electronically along with the student 
transcripts? 


a. Eighth-grade completion status (i.e., a variable that indicates whether or not the student 
advanced from eighth grade at the end of the 2016-2017 school year) 
Yes O No 


b. The number of days the student was absent in sixth, seventh, and eighth grades (reported 
by individual grade level, not a combined count) 
Yes O No 


c. Student transfer status (i.e., a variable that indicates whether or not the student transferred 
into the district in sixth, seventh, or eighth grade) 
Yes O No 


d. The number of course credits the student earned in sixth, seventh, and eighth grades 
(reported by individual grade level, not a combined count) 
Yes oO No 


e. The grade point average the student earned in sixth, seventh, and eighth grades (reported 
by individual grade level, not a combined count) 
Yes O No 


f. The scores on any state or district assessments the student took in sixth, seventh, and 
eighth grades 
Yes oO No 


If you can only provide partial data for any of the elements above (e.g., a combined count of absent 
days or the grade point averages only for seventh and eighth grades) or can explain why any of the 
elements above cannot be provided, please note it in the field on the top of the next page. 
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Finally, we will ask you some questions about collecting the catalog, transcript, and other student 
information discussed above. 


If you indicated in Question 1 that you can provide an electronic course catalog (or course lists) to us, 
please complete the questions below about the format of the course catalog and how to send it. If you 
indicated in Question 1 that an online course catalog for sixth through eighth grades was available, 
please skip to Question 17. 


15. In what format is the course catalog stored; for example, a Microsoft Word file, a Microsoft Excel 
workbook, ASCII data files, rich text format (RTF) files, PDF files, or some other format? 


16. Will you be encrypting the file containing the course catalog (i.e., a password-protected PDF, a 
zipped file with a security password)? 
Yes Oo No 


If you answered Yes above, how will the file will be encrypted? 


Please send the password used to encrypt the course catalog file via email to Westat at 


MSTS@westat.com. 
Questions 17 through 20 ask about the format of the student transcripts and how they will be sent. 


17. Are the electronic transcript data managed and stored within the school district’s data center, or 
does an outside vendor manage the district’s electronic transcripts? 


School district data center 
Outside vendor 


a. If you answered Outside vendor, are you the person to contact the vendor to arrange the 
electronic transfer of the transcripts? 
Yes Oo No 


b. If you are not the contact person, please list the name, telephone number, and email 
address of the person we would need to contact. 
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18. In what format are the electronic transcript data files stored; for example, Microsoft Word files, 
Microsoft Excel workbooks, Microsoft Access databases, SAS databases, SPSS databases, ASCII data 
files, rich text format (RTF) files, PDF files, or some other format? 


19. Is there personal information about the students listed on the electronic transcripts, such as student 
name or address? 
Yes Oo No 


a. If you answered Yes, Westat can remove this personal identifying information upon 
receiving the transcripts, or you (or your vendor) can remove this information before 
sending the transcripts. Which process of removing the student personal identifying 
information would you prefer? 


Westat removes the data 
School district/vendor removes the data 


20. Will you be encrypting the file containing the transcript data (i.e., a password-protected Excel file, a 
zipped file with a security password)? 
Yes Oo No 


If you answered Yes above, how will the file will be encrypted? 


Please send the password used to encrypt the student transcript file via email to Westat at 


MSTS@westat.com. 


Questions 21 and 22 ask about the format of the additional student information (as outlined in 
Question 14) that needs to be sent. 


21. In what format will this additional student information be stored; for example, a Microsoft Word 
file, a Microsoft Excel workbook, ASCII data files, rich text format (RTF) files, PDF files, or some other 
format? 
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22. Will you be encrypting the file containing the additional student information (i.e., a password- 
protected Excel file, a zipped file with a security password)? 
Yes Oo No 


If you answered Yes above, how will the file will be encrypted? 


Please send the password used to encrypt the student information file via email to Westat at 


MSTS@westat.com. 


Thank you. Westat will set up a secure FTP site for your school district where the course catalog, student 
transcripts, and additional student information should be uploaded. A follow-up email that contains the 
web address for your FTP site will be sent to the TUDA coordinator or the contact person the 
coordinator designates. 


Thank you for completing the questionnaire. Be sure you save your responses to the form before 
exiting, otherwise your responses will be lost. Please email the completed form to the MSTS help desk 
at MSTS@westat.com. If you have any questions, please feel free to contact the MSTS help desk at 
the email address above. 
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Appendix F—List of Request Data 
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TUDA educational information 


Eighth-grade graduation requirements 


Credits earned for completing a year-long course 


Common grading standard (for course grade standardization) 


State or district assessments students take in grades 6—8 (if any) 


District-wide educational programs available to students in grades 6—8 


TUDA district catalog 


Course name 


Course ID number (linkable to transcripts) 


Course description (if available) 


Credits earned for passing the course 


Special education course designation 


English Language Learner course designation 


Online course designation 


Student ID and demographic information 


Unique Student Identifier (to match with NAEP) 
NAEP assessment student took (will get from NAEP) 


Month of birth (will get from NAEP) 


Year of birth (will get from NAEP) 
Gender (will get from NAEP) 


Race/ethnicity (will get from NAEP) 


NSLP status (will get from NAEP) 
IEP/ELL status (will get from NAEP) 


Student educational information 


Eighth-grade graduation status (yes/no) 
Number of days absent (sixth, seventh, eighth grade) 


Student transfer into TUDA between grades 6-8? (yes/no) 


Number of credits earned (sixth, seventh, eighth grade) 
Grade point average (sixth, seventh, eighth grade) 


State and/or district assessment scores (if requested) 


Student course records data 


Course name 


Course ID number (linkable to catalog) 


Grade level taken 


School year taken 


Length of course (semester, quarter, etc.) 


Number of credits earned 


Grade earned (letter or numeric) 


Level of course (regular, honors, etc.) 


Location, if not taught at school 


Special education course designation 


English Language Learner course designation 


Transfer course designation 
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Appendix G—NCES Affidavit of Nondisclosure 


NCES Affidavit of Nondisclosure 


Affidavit of Nondisclosute 


(Job Title) (Date Assigned to Work with NCES Data) 
(Organization, State or Local Agency Name) 


(Organization or Agency Address) 
(NCES Database or File Containing Individually Identifiable Information”) 


I, , do solemnly swear (or affirm) that when given access to the 
subject NCES database or file, I will not - 


(i) use or reveal any individually identifiable information furnished, acquired, retrieved, or assembled by me 
or others, under the provisions of Section 183 of the Education Sciences Reform Act of 2002 (P.L. 107-279) 
and Title V, subtitle A of the E-Government Act of 2002 (P.L. 107-347) for any purpose other than 
statistical purposes specified in the NCES survey, project, or contract; (it) make any disclosure or 
publication whereby a sample unit or survey respondent (including students and schools) could be identified 
or the data furnished by or related to any particular person or school under these sections could be 
identified; or (11) permit anyone other than the individuals authorized by the Commissioner of the National 
Center for Education Statistics to examine the individual reports. 


(Signature) 


[The penalty for unlawful disclosure is a fine of not more than $250,000 (under 18 U.S.C. 3571) or 
imprisonment for not more than five years (under 18 U.S.C. 3559), or both. The word “sweat” should be 
stricken out when a person elects to affirm the affidavit rather than to swear to it.] 


City/County of Commonwealth/State of 
Sworn to and subscribed before me this day of 
, 20 . Witness my hand and official Seal. 


(Notary Public/Seal) My commission expires 
* Request all subsequent follow-up data that may be needed. This form cannot be amended by NCES, so 


access to databases not listed will require submitting additional notarized Affidavits. 
Form last revised 02/08/07 
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